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1 Introduction 

Environment maps are used to simulate reflections off curved ob- 
jects. We present a technique to reflect a user, or a group of users, in 
a real environment, onto a virtual object, in a virtual reality applica- 
tion, using the live video feeds from a set of cameras, in real-time. 
Our setup can be used in a variety of environments ranging from 
outdoor or indoor scenes. 

The need for such a technique becomes obvious under the defini- 
tion of replacement reality: the ability to interact with highly credi- 
ble digital models in a natural and intuitive manner, unencumbered 
by intrusive equipment, using your senses of touch and hearing, as 
well as vision, offering the user an enhanced and personalised im- 
mersive experience. Consider a virtual car showroom. Confronting 
a user with a virtual customization toolset for a car, providing a 
high-fidelity virtual representation, enhanced by a sense of immer- 
sion including both surrounding environment and the user within 
the same scene, is an invaluable marketing tool. 


2 Embedded Reflection Mapping 

Environment maps are textures that describe, for all directions, the 
incoming or outgoing light at one point in space. The implicit as- 
sumption, with environment maps, that the incoming illumination 
on a point in a surface re ally depends only on the direction of the 
reflected ray is still valid | Olano et al. 2002 1. 

The mirror reflection term, described by an environment map, 
can be used in combination with local illumination terms. By com- 
bining high-dynamic range images and a filtered versi on of the en- 
vironment map, lighting effects as described by | Debevec 1998 1 are 
possible. 

The specular term of a lighting model can be factored in a stan- 
dard environment map and a fresnel term. Our environment map is 
achieved by compositing of a real scene environment map and an 
environment map with the user, or group of users. 

Under this factorization, we wish to further factor our environ- 
ment map. Our standard environment map will be a composition of 
an environment map of a real scene and an environment map con- 
taining only the user or group of users, lit as they would stand in 
the real scene environment, and generated using the same camera 
parameters of our real scene environment, in real-time. 

Environment maps represent directionally dependent informa- 
tion. The mapping between directions, as encoded in the map, and 
texture coordinates is the parameterization of the environment map. 

The parameterization we have chosen is the spherical environ- 
ment map. The light information on a spherical environment map 
for a certain point is all the visible light from that given point in 
space projected onto the inside surface located concentrically sur- 
rounding that point. The light information can then be encoded onto 
a two-dimensional angular plane using the discretized form of the 
parametric equation of the sphere. 

In order to build such an environment map, the desired capture 
point, must be captured along different directions. These images are 


then warped onto a sphere along the corresponding directions. The 
warping is done by sampling the sphere along all of the directions 
and classifying the pixels according to each of the directions. 



Figure 1 : User being reflected onto a virtual wheel. The black spots 
are part of the geometry of the wheel. The lightprobe on the wheel 
is courtesy of Paul Debevec. 


Our camera setup follows closely that of I De bevec et al. 2002 1. 
Color and infrared images of the user or group of users are cap- 
tured and the live video feeds are then streamed over to a PC server. 
After the mattes are acquired, the composite environment map is 
generated and modulated with a Fresnel term to achieve the final 
reflection onto the virtual object. 

3 Conclusion 

We have described a technique to reflect a user or group of users, 
dynamically, onto a virtual object. An example of a virtual reality 
application in which the technique is useful has been presented. 

It is the authors intent to combine this technique with environ- 
ment map prefiltering in order to improve the visual quality of the 
resulting image. 
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